BMW Group and Meta’s Reality Labs present joint research for interlinking extended reality devices with the digital vehicle ecosystem.

Munich / Mountain View . The BMW Group Technology Office USA in Mountain View presents a research update with Meta’s Reality Labs Research, demonstrating, for the first time, the ability to accurately display stable virtual reality (VR) and mixed reality (MR) content to passengers in a fast-moving car, even when making turns, going over speed bumps, and accelerating. By integrating Meta Quest’s tracking system with BMW’s sensor data, the researchers enabled a series of stable “car-locked” gaming, entertainment, productivity, and meditation experiences.

Claus Dorrer, Head of BMW Group Technology Office USA in Mountain View: “It is too early to tell exactly how or when this technology will make it into customers’ hands, but we envision a number of potential use cases for XR devices in vehicles — from assisting the driver in locating their car in a crowded parking lot to alerting them to hazards on the road and surfacing important information about the vehicle’s condition. The implications of future AR glasses and VR devices — for passengers as well as drivers — are promising. The research partnership with Meta will allow us to discover what immersive, in-vehicle XR experiences could look like in the future and spearhead the seamless integration of such devices into cars.”

Proof-of-concept prototype overcomes key technical challenges.
VR headsets use cameras and motion sensors to understand their exact position, so when the user moves their head or looks around, the virtual content remains stable. However, in fast-moving vehicles, the headset’s sensors get confused by conflicting signals from the device’s movement within the car and the vehicle’s movement in the world. Simply put, the headset’s motion sensors think it is moving, but its cameras think it is standing still when looking at the car’s interior.

That mismatch means today’s VR headsets can’t display stable virtual content when travelling inside a fast-moving vehicle, particularly if the vehicle accelerates, turns fast, or encounters bumps on the road.

To solve this challenge, BMW and Meta researchers collaborated to incorporate information from a BMW car’s sensor array into the tracking system of a Meta Quest Pro. That allowed the tracking system to simultaneously calculate the device’s location relative to the car and the moving car’s location relative to the world, enabling it to accurately anchor virtual objects to a moving car and demo compelling VR and MR passenger experiences.

Richard Newcombe, Vice President of Research Science, Reality Labs Research: “Our research prototype shows that we can enable entertaining and comfortable passenger experiences that are anchored to the car itself, including VR and MR gaming, entertainment, productivity, and even meditation capabilities. The technology has the potential to transform how we can safely interact with our environment while traveling, and as we progress into reliable world-locked content on the road to AR glasses, we’re hopeful it will be possible for passengers to see things like markers for landmarks, restaurants, places of interest, and more.”

Combining the movement data generated by the vehicle with XR headset sensor capabilities allowed us to subtract the vehicle motion and more accurately track the headset’s movement in a vehicle. That in turn enabled us to render content which is synced to the movement of the vehicle in real-time, both inside and outside of the vehicle at high speeds, regardless of whether the vehicle is on a flat, straight street or a bumpy road with sharp turns.

This proof-of-concept prototype demonstrates an in-car full range of a person’s movements (known as six degrees of freedom) and an XR experience that simultaneously calculates a headset’s location relative to the car and the moving car’s location relative to the world by integrating with the car’s sensors.

The BMW Group has been successful in pioneering digital technologies like vehicle access based on the Digital Key and in driving industry standardization for such innovations. The company’s global network of technology offices has been crucial in kickstarting such innovation projects. The BMW Group is open to including additional partners in order to achieve the goal of creating an industry standard for connecting XR consumer electronics devices to vehicles. For potential series development, the BMW Group is committed to the privacy-by-design principle. This means, in all digital feature development, the BMW Group takes steps to ensure that customer data is protected and securely processed in accordance with data privacy requirements by means of established processes — across all the markets in which the company operates.

Link to Meta Reality Labs information on the project.